pullbacks don't work. i wasnt actually doing pullbacks this entire time. $U^{T}\hat{v}_i$ is not a pullback. it doesnt represent any useful computation that the model is doing, unlike transcoders which approximate MLPs and are supposed to be a replacable computation component. A huge conceit of DCTs is that they are supposed to find elicitative features much more efficiently and mechanistically than with SAEs etc. they cannot reconstruct computations or anything. i thought a dot product of a tranposed matrix by a vector was always a pullback. well now i dont know what to do. there is no other conceivable operation i can think of that can find 'model-native' circuits and ive done a lot of drooling on what are now entirely hypothetical scenarios where you magically find model circuits without reconstruction.

w.r.t to the entire persistent shallow circuit hypothesis... all i had to do was read the rest of the post to find this

> the lower jailbreak scores in this table might indicate that the $\hat{u}_ℓ$'s don't perfectly align with the core "should refuse request" direction, but rather capture some downstream effects of this direction. This doesn't necessarily contradict the "persistent shallow circuits" hypothesis; it only implies that we should be cautious about assuming the DCT will learn some specific "high-level action" feature in the target layer (as I did for illustrative purposes in table 1), as opposed to some side effect

it looks like i have found that these are not exactly feature circuits! i will need to take some different approaches

- [averaging many output directions](https://www.lesswrong.com/posts/fSRg5qs9TPbNy3sm5/deep-causal-transcoding-a-framework-for-mechanistically#Ablating__averaged__target_layer_features_also_works) tends to produce a "feature." the standard learning objective for exponential DCTs doesnt do this though.
	- one way to workaround is to just enforce/regularize as many output directions to be features. do some sort of optimization where you take advantage of all the output feature multiplicity(?) like by grouping up and pruning candidate output factors, or doing it on a secondary network and compare objective values. this could be annoying, but nice to have, cus much more interpretability
	- another way: notice the factors alone do not have to form a computational graph, and that doing smth like EAP modified to take averages of output features, and then using the causal part of the objective as a metric, is ok
- without anything resembling pullbacks working, having to rank factors by logits to determine edges is quite inconvenient. and this approach hardly captures feature interactions either, exp DCTs are not a linear mapping $\hat{v}_l \rightarrow \hat{u}_l$. in this case, taking the former route should help here a lot, but at that point, you might as well not bother with the pullback
- alternatively... you might be able to take a pullback (or more appropriately, evaluate an attribution score) by exploiting the fact that factors are derived through a training process that approximates a one-hidden-layer exponential MLP (which it does seemingly quite well). you can only attribute the effect of one feature $\hat{v}_i$ to one output direction $\hat{u}_j$, but you can easily enumerate every $\hat{v}$ and then apply EAP for more complex edges